The Holy Grail of Content-Based

multimedia and—image, video, audio, graphics, text, and any number of combinations—has increased in the last decade. We’ve seen a common theme of developing automatic analysis techniques for deriving metadata (data describing information in the content at both syntactic and semantic levels). Such metadata facilitates developing innovative tools and systems for multimedia information retrieval, summarization, delivery, and manipulation. Many interesting demonstrations of potential applications and services have emerged—finding images visually similar to a chosen picture (or sketch); summarizing videos with thumbnails of keyframes; finding video clips of a specific event, story, or person; and producing a two-minute skim of an hour-long program. (Audio–visual skims are condensed media clips that summarize information in the content.) There’s much excitement and buzz created by these fancy applications. But people are always asking, What will be engineering’s Holy Grail for content-based media analysis in practical applications? My response is that the answer is complex. It’s less about a specific algorithm or service than a rigorous methodology to formulate and evaluate content-based analysis research.